decred.org/dcrdex@v1.0.5/docs/wiki/Getting Started With Fuzzing.md (about)

     1  # Getting Started With Fuzzing
     2  
     3  Go started supporting fuzzing in its standard toolchain [beginning in Go 1.18](https://go.dev/doc/fuzz/). Fuzzing is a type of automated testing which continuously manipulates inputs to a program to find issues such as panics or bugs. These semi-random data mutations can discover new code coverage that existing unit tests may miss, and uncover edge case bugs which would otherwise go unnoticed. Since fuzzing can reach these edge cases, fuzz testing is particularly valuable for finding security exploits and vulnerabilities.
     4  
     5  ## Writing a Fuzz Test
     6  
     7  A fuzz test **must** be in a \*_test.go file as a function in the form FuzzXxx. This function must be passed a*testing.F argument, much like a `*testing.T` argument is passed to a TestXxx function.
     8  
     9  Below is an example of a fuzz test that’s testing the behaviour of `dex/encode/encrypt`.
    10  
    11  ```go
    12  // Fuzz name must be `FuzzXXX` which accepts only a `*testing.F`, and has no return value.
    13  func FuzzDecrypt(f *testing.F) {
    14  
    15  	// Providing seed corpus helps the fuzzing engine to generate new seeds efficiently.
    16  	seeds := []struct {
    17  		b []byte
    18  		n int
    19  	}{{
    20  		n: 200,
    21  		b: []byte("4kliaOCha2longerbyte"),
    22  	}, {
    23  		n: 20,
    24  		b: []byte("short123456"),
    25  	}, {
    26  		n: 50,
    27  		b: []byte("23Fgfge34"),
    28  	}, {
    29  		n: 10,
    30  		b: []byte("asdf$#@*(gth#4"),
    31  	}}
    32  
    33  	for _, seed := range seeds {
    34  		f.Add(seed.n, seed.b) // Use f.Add to add the seed corpus in the same order as the fuzz target arguments.
    35  	}
    36  
    37  	// A fuzz target must be a function without a return value, which accepts a
    38  	// `*testing.T` as the first parameter, followed by the fuzzing arguments.
    39  	// There must be exactly one fuzz target per fuzz test.
    40  
    41  	// `f.Fuzz` accepts a fuzz target and runs fuzzing with randomly generated values
    42  	// based on the seed corpus provided. All seed corpus entries must have types
    43  	// which are identical to the fuzzing arguments, in the same order. This is true
    44  	// for calls to `(*testing.F).Add`.
    45  	f.Fuzz(func(t *testing.T, n int, b []byte) {
    46  		// Inputs can be validated and skipped if they are not suitable.
    47  		// This test will panic if len(b) or n is greater than encode.MaxDataLen.
    48  		if n < 1 || n > encode.MaxDataLen || len(b) > encode.MaxDataLen {
    49  			t.Skip()
    50  		}
    51  		crypter := NewCrypter(b)
    52  		thing := randB(n)
    53  		encThing, err := crypter.Encrypt(thing)
    54  		if err != nil {
    55  			t.Fatalf("Encrypt error: %v", err)
    56  		}
    57  		reThing, err := crypter.Decrypt(encThing)
    58  		if err != nil {
    59  			t.Fatalf("Decrypt error: %v", err)
    60  		}
    61  		if !bytes.Equal(thing, reThing) {
    62  			t.Fatalf("%x != %x", thing, reThing)
    63  		}
    64  	})
    65  }
    66  ```
    67  
    68  The fuzzing arguments can only be the following types:
    69  
    70  - string, []byte
    71  - int, int8, int16, int32/rune, int64
    72  - uint, uint8/byte, uint16, uint32, uint64
    73  - float32, float64
    74  - bool
    75  
    76  ## Running Fuzz Tests
    77  
    78   There are two modes of running fuzz tests:
    79  
    80  - As a unit test (default `go test`).
    81      Fuzz tests are run much like a unit test by default. Each seed corpus entry will be tested against the fuzz target, reporting any failures before exiting. Failed Fuzz inputs generated by the fuzzing engine for that particular fuzz target are re-run against the fuzz target as part of the default `go test`.
    82  
    83  - With fuzzing `go test -fuzz=FuzzTestName`.
    84      Each seed corpus entry will be tested against the fuzz target. Also, If there are seed corpus already generated for the fuzz target in `$GOCACHE` they will be run against the fuzz target, and then the fuzzing engine will continue to generate and test seed corpus until `-fuzztime`. See [Fuzz Flags](#other-highlighted-fuzz-flags).
    85  
    86  To enable fuzzing, run go test with the `-fuzz` flag, providing a test name (e.g `FuzzTestName`) or regex matching a **single** fuzz test. By default, all other tests in that package will run before fuzzing begins. This is to ensure that fuzzing won’t report any issues that would already be caught by an existing test.
    87  
    88  **Note**: Fuzzing cannot be run for multiple packages at the same time using the `-fuzz` flag. You **must** specify **at most one Fuzz test** (e.g `-fuzz=FuzzXXX`) [See: [support for multiple Fuzz tests](https://go.googlesource.com/proposal/+/master/design/draft-fuzzing.md#fuzzing-engine-supports-multiple-fuzz-tests-at-once)].
    89  
    90  ## Command Line Output
    91  
    92  - **elapsed**: the amount of time that has elapsed since the process began
    93  - **execs**: the total number of inputs that have been run against the fuzz target (with an average execs/sec since the last log line)
    94  - **new interesting**: the total number of “interesting” inputs that have been added to the generated corpus during this fuzzing execution (with the total size of the entire corpus)
    95  
    96  ```text
    97  $ go test -v ./... --fuzz=FuzzDecrypt --fuzztime=10x 
    98  === RUN   TestDecrypt
    99  --- PASS: TestDecrypt (0.04s)
   100  === RUN   TestSerialize
   101  --- PASS: TestSerialize (0.31s)
   102  === RUN   TestRandomness
   103  --- PASS: TestRandomness (0.08s)
   104  === FUZZ  FuzzDecrypt
   105  fuzz: elapsed: 0s, gathering baseline coverage: 0/7 completed
   106  fuzz: elapsed: 0s, gathering baseline coverage: 7/7 completed, now fuzzing with 8 workers
   107  fuzz: elapsed: 1s, execs: 10 (18/sec), new interesting: 2 (total: 9)
   108  --- PASS: FuzzDecrypt (0.57s)
   109  PASS
   110  ok      decred.org/dcrdex/dex/encrypt   1.172s
   111  ```
   112  
   113  ## Failing Input
   114  
   115  Failing inputs generated by the fuzzing engine are saved to the main directory of the project (e.g `dcrdex/testData/fuzz/$fuzzTarget`), and are re-run as part of `go test` (without `-fuzz FuzzTarget`) for `$fuzzTarget`.
   116  
   117  A failure may occur while fuzzing for several reasons:
   118  
   119  - A panic occurred in the code or the test.
   120  - The fuzz target called t.Fail, either directly or through methods such as t.Error or t.Fatal.
   121  - A non-recoverable error occurred, such as an os.Exit or stack overflow.
   122  - The fuzz target took too long to complete. Currently, the `timeout for an execution of a fuzz target is 1 second`. This may fail due to a deadlock or infinite loop, or from intended behaviour in the code.
   123  
   124  ## Other Highlighted Fuzz Flags
   125  
   126  - **-fuzztime**:
   127      The total time or number of iterations that the fuzz target will be executed before exiting, specified as a time.Duration (for example, `-fuzztime 1h30s`) or using a special syntax Nx to run the fuzz target N times (for example, `-fuzztime 1000x`). The default is to run forever.
   128  - **-fuzzminimizetime**: the time or number of iterations that the fuzz target will be executed during each minimization attempt, default 60sec. You can completely disable minimization by setting -fuzzminimizetime 0 when fuzzing.
   129  - **-parallel**: the number of fuzzing processes running at once, default $GOMAXPROCS. Currently, setting -cpu during fuzzing has no effect.
   130  - **-keepfuzzing**: Keep running the fuzz test if a crasher is found. (default false)
   131  
   132  ## Best Practices
   133  
   134  - Fuzz targets `should be fast and deterministic` so the fuzzing engine can work efficiently, and new failures and code coverage can be easily reproduced.
   135   Fuzzing is most effective with the most primitive methods and functions. As such, this will take some careful targeting and potentially some very light refactoring here and there to get fuzzable functions.
   136  - Since the fuzz target is invoked in parallel across multiple workers and in nondeterministic order, the state of a fuzz target should not persist past the end of each call, and the behaviour of a fuzz target `should not depend on a global state`.
   137  - Keeping auto-generated seed corpus in $GOCACHE increases the efficiency of the fuzzing engine. The more seed corpus the better.
   138  - Provide sufficient seed corpus. This could be in the test file, another file or a directory. This will be a guide to the fuzzing engine in generating sensible random inputs.
   139  To convert seed corpus from another file or directory to Go fuzzing corpus format use [file2fuzz](https://pkg.go.dev/golang.org/x/tools/cmd/file2fuzz):
   140  
   141    ```sh
   142  
   143  	$ go install golang.org/x/tools/cmd/file2fuzz@latest
   144  	$ file2fuzz
   145  
   146    ```
   147  
   148  ## Clean-up
   149  
   150  - Fuzz tests can do clean-up using `f.Clean(f func())`.
   151  - Use `go clean -fuzzcache` to remove files stored in the Go build cache for fuzz testing aka `seed corpus`. The fuzzing engine caches seed corpus files that expand code coverage, so removing these `seed corpus` may make fuzzing `less effective` until new inputs are found that provide the same coverage. These files are distinct from those stored in `testData` directory i.e `dcrdex/testData/fuzz` dir; clean does not remove those files.
   152  
   153  ## References
   154  
   155  - [Go Fuzz Doc](https://go.dev/doc/fuzz/)
   156  - [Go Fuzz Official Tutorial](https://go.dev/doc/tutorial/fuzz)
   157  - [Go Fuzz Design Draft](https://go.googlesource.com/proposal/+/master/design/draft-fuzzing.md)
   158  - [A Good Fuzz Target](https://github.com/google/fuzzing/blob/master/docs/good-fuzz-target.md)
   159  - [Glossary](https://go.dev/doc/fuzz/#Glossary:~:text=Proposal-,Glossary,-corpus%20entry%3A)